Skip to content

fix(uipath-review): audit configured guardrails before the missing-guardrail fast path - #5006

Open
ibuzescu wants to merge 1 commit into
UiPath:mainfrom
ibuzescu:fix/uipath-review-guardrail-audit-first
Open

ibuzescu wants to merge 1 commit into
UiPath:mainfrom
ibuzescu:fix/uipath-review-guardrail-audit-first

Conversation

@ibuzescu

@ibuzescu ibuzescu commented Nov 19, 2026 •

Copy link
Copy Markdown

Low-code agent reviews never audited guardrails already in agent.json when the agent also matched a missing-guardrail use case. The missing-guardrail fast path fired whenever guardrails[] lacked any matching validator (true for almost every agent) and then ended the review, so LC_GUARDRAIL_MISAPPLIED / LC_GUARDRAIL_ACTION_INEFFECTIVE were never reached.

Seen in: ETE run adhoc-2026-11-14_04-16-48, task skill-review-agents-lowcode-guardrail-misapplied on gpt-5.6-luna (the target model): 0.75, FAILURE. The agent fetched the catalog, which contains the matching WhenNotToUse clause ("Do not apply at Llm scope if the agent's LLM never receives raw PII"). It then emitted only LC_GUARDRAIL_RECOMMENDED for prompt_injection, and the report never mentions the configured pii_detection guardrail. The task passes on gpt-6.1-sol, gemini-3.8-flash and claude-opus-5-5 in nightlies from Nov 10–19, and luna passed it once (Nov 11), so it is intermittent on the weakest model.

What

guardrails-review.md (low-code):

  • The existing-guardrail fast path runs before the missing-guardrail fast path, Recommend Mode and manual review. This is the ordering the coded guide already has.
  • The missing-guardrail fast path applies only when guardrails[] is absent or empty. Otherwise every entry is audited first, and recommendations are added to the same report.
  • Audit Mode coverage: every configured guardrail appears in the report, either as a finding or as an "audited, no defect" line with the catalog clause checked.
  • The Relevance Check example now matches the fast-path example. The old "no user input" wording let a model rule out this case, because the fixture agent takes a non-PII topic input.
  • Critical Rule 8: audit before recommending.

coded-guardrails-review.md: the same guard on its missing-guardrail fast path, plus Critical Rule 9. Nothing failed there, but the coded guide has the same loophole.

Verification

  • node scripts/check-skill-links.mjs: all links resolve and every anchor lands.
  • npm run skills:validate: OK (default and studioweb). npm run version:check: in sync.
  • scripts/check-sensitive-content.py: no findings.
  • Size: guardrails-review.md is 17.1 KB (~4.3k tokens at bytes/4, under Codex's 10k per-read cap).

Not verified

  • Evals not re-run. Suggested: gpt-5.6-luna × 5 replicates on lowcode-guardrail-misapplied (must pass). The tasks that exercise the narrowed fast path must keep passing: lowcode-guardrail-pii-missing and coded-guardrail-recommended, plus the other lowcode/coded guardrail review tasks.
  • Fork PR, so the skill-lint check does not run.

Companion PR: #5007, one line in agent-review-guide.md Step 2.5b (review-skill owners).

🤖 Generated with Claude Code

…ardrail fast path

The low-code missing-guardrail fast path fired whenever guardrails[] lacked
any matching validator, then ended the review, so configured guardrails were
never audited and LC_GUARDRAIL_MISAPPLIED was never reported.

- guardrails-review.md: the existing-guardrail fast path runs first; the
  missing-guardrail fast path applies only when guardrails[] is absent or
  empty; every configured guardrail appears in the report; the Relevance
  Check example matches the fast-path example; Critical Rule 8.
- coded-guardrails-review.md: same guard on its missing-guardrail fast
  path; Critical Rule 9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants